Evaluating AI Usage for Evaluation Purpose

“We are drowning in information, while starving for wisdom. The world henceforth will be run by synthesizers, people able to put together the right information at the right time, think critically about it, and make important choices wisely.”

Edward Osborne Wilson

Current Challenge

Considering the number of published evaluation reports across the UN system, information retrieval and evidence generalization challenges have arisen.

How to extract the most relevant findings and recommendations from within a specific context and to reuse and re-inject them in a different but appropriate context?

The fifth wave of the evidence revolution will be triggered by AI

The four waves of the evidence revolution, published in Nature, Howard White, 2019

Results Cherry Picking: how to build effective “Evaluation Brief”?

Choosing what to include and what to exclude, especially in terms of highlighting critical aspects while deciding on what are the less relevant details to omit…

Relying on automated retrieval can help improving the objectivity and independence of the evaluation report summarization.

Retrieval Augmented Generation (RAG) at Rescue

Builing a RAG Pipeline requires test & evaluation

A RAG Evaluation Framework

Applying Data Science Approach - documentation & reproducibility

The next stage: Human-in-the-loop

AI Deployment: Buy or Build

Presentation title

Left column

Right column